4 research outputs found
Multimedia Semantic Integrity Assessment Using Joint Embedding Of Images And Text
Real world multimedia data is often composed of multiple modalities such as
an image or a video with associated text (e.g. captions, user comments, etc.)
and metadata. Such multimodal data packages are prone to manipulations, where a
subset of these modalities can be altered to misrepresent or repurpose data
packages, with possible malicious intent. It is, therefore, important to
develop methods to assess or verify the integrity of these multimedia packages.
Using computer vision and natural language processing methods to directly
compare the image (or video) and the associated caption to verify the integrity
of a media package is only possible for a limited set of objects and scenes. In
this paper, we present a novel deep learning-based approach for assessing the
semantic integrity of multimedia packages containing images and captions, using
a reference set of multimedia packages. We construct a joint embedding of
images and captions with deep multimodal representation learning on the
reference dataset in a framework that also provides image-caption consistency
scores (ICCSs). The integrity of query media packages is assessed as the
inlierness of the query ICCSs with respect to the reference dataset. We present
the MultimodAl Information Manipulation dataset (MAIM), a new dataset of media
packages from Flickr, which we make available to the research community. We use
both the newly created dataset as well as Flickr30K and MS COCO datasets to
quantitatively evaluate our proposed approach. The reference dataset does not
contain unmanipulated versions of tampered query packages. Our method is able
to achieve F1 scores of 0.75, 0.89 and 0.94 on MAIM, Flickr30K and MS COCO,
respectively, for detecting semantically incoherent media packages.Comment: *Ayush Jaiswal and Ekraam Sabir contributed equally to the work in
this pape
COMPENSATED SIGNATURE EMBEDDING BASED MULTIMEDIA CONTENT AUTHENTICATION SYSTEM
Digital content authentication and preservation is an extremely challenging task in realizing decentralized digital libraries. The concept of compensated signature embedding is proposed to develop an effective multimedia content authentication system. The proposed system does not require any third party reference or side information. Towards this end, a content-based fragile signature is derived and embedded into the media using a robust watermarking technique. Since the embedding process introduces distortion in the media, it may lead to authentication failure. We propose to adjust the media samples iteratively or using a closed form process to compensate for the embedding distortion. Using an example image authentication system, we show that the proposed scheme is highly effective in detecting even minor modifications to the media. Index Terms — authentication, preservation, watermarking, compensated signature embeddin